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ABSTRACT 



Since 1991, the National Science Foundation has funded 59 
state, urban, and rural systemic initiatives to promote achievement in math, 
science, and technology. The Appalachian Rural Systemic Initiative (ARSI) is 
a six-state consortium that focuses these efforts on low- income rural 
schools. A primary means of accomplishing this is a 1-day, one-school site 
visit, called a Program Improvement Review, done by an ARSI math or science 
expert. A first-hand account of the Program Improvement Review by a former 
ARSI expert leads to the following observations: ARSI experts construe the 
process of educational achievement as a thoroughly understood, relatively 
simple mechanism manifest in static indicators of school effectiveness. In 
consequence, ARSI has standardized and accelerated its Program Improvement 
Review process through excessively rout ini zed observation based on short-cut 
procedures and unvalidated instruments. ARSI experts take their evaluation 
criteria for granted as embodying the one right way to teach math and science 
anywhere. Student responses and student-teacher interactions are irrelevant 
to ARSI evaluations. Departures from ARSI criteria, even in the presence of 
overwhelmingly favorable student performance, are negatively sanctioned. 
Premised on distortions and oversimplifications, the Reviews become 
caricatures of early 20th-century Taylorism, and are likely to undercut 
achievement rather than enhance it. (Contains 20 references.) (SV) 
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QUICK-AND-DIRTY REFORM IN LOW- INCOME RURAL SCHOOLS 
NSF'S APPALACHIAN RURAL SYSTEMIC INITIATIVE 



Since 1991, the National Science Foundation has funded 

fifty-nine state, urban, and rural systemic initiatives. 

The purpose of the initiatives is to promote achievement in 

% 

math, science, and technology among all students, and ^o 
encourage schools and communities to secure the resources 
needed to maintain such outcomes . The Appalachian RuraijL 
Systemic Initiative (ARSI) is a six- state consortium which 
focuses these efforts on low-income, rural schools. A^ 
primary means of accomplishing this is a one-day-one-school 
site visit, called a Program Improvement Review, done by an 
ARSI math or science expert. In practice, the Program 
Improvement Reviews are premised on distortions and oyer- 
sinplifications as to the static, easy-to-understand and 
easy-to-evaluate nature of educational achievement in rural 
Appalachian schools. As a result, the Reviews become 
caricatures of early- twentieth century Taylorism, and ^re 
likely to undercut achievement in science and math rather 
than enhance it . This research is based on participation- 
observation by one who was first an ARSI expert -aspirant, 
then a newly-minted ARSI expert, and finally an A|iSI 
dropout . 
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Efforts to promote economic development and eliminate 
poverty through investment in public education have a l^ng 
history in the U.S. (see, for example, Bowles and Gintis, 
1976; Kaestle, 1983; Perkinson, 1995; Spring, 1997;^ 

McMurrer and Sawhill, 1998) . In recent years, such efforts 
have included special attention to elementary and secondary 
schooling in science, math, and technology (Ashton and 
Sung, 1997; Senate Committee on Labor and Human Resourq^es, 
1997) . 



This ettphasis is premised on the assumption that in an 
increasingly science -based, technology- intensive world, the 
economic well-being -- perhaps even the simple survival -- 
of individuals and entire societies requires ever-higher 
levels of pure and applied scientific and mathematical 
knowledge (Shapiro and Varian, 1998; National Council of 
Teachers of Mathematics, 1998; Reich, 1992). 

The National Science Foundation's Systemic Initiatives 

In line with this world-view, since 1991 the National 
Science Foundation has funded fifty-nine state, urban, and 
rural systemic initiatives (National Science Foundat:^on, 
1999) . The purpose of each systemic initiative is to 
promote education in math, science, and technolcpgy 
(National Science Foundation, 1994a) . 
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Published literature on the initiatives is hard to 
find, and unpublished technical reports are not numercjus. 
The origin of the term "systemic initiative" remains 
unclear. This terminology may follow, however, from NSjF's 
judgment that education involves entire communities 
(Shields, 1997) . At its best, in this view, education in 
math and science focuses on everyday applications in 
communities where schools are located (National Sciepce 
Fovuidation, 1994b) . The communities themselves, in a 
reciprocal process, benefit from development of a 
technologically literate workforce (Consortium for Policy 
Research in Education, 1995) . 

NSF'S Appalachian Rural Systemic Initiative (ARSI) 

The Appalachian Rural Systemic Initiative, or ARSI, is 
a six-state consortium, covering all of the Appalachian 
region of the U.S (Harmon and Blanton, 1997) . Consistent 
with NSF'S intent, ARSI's ambitious objective is to 
facilitate educational change in economically disadvantaged 
rural schools, resulting in high achievement of all 
students in mathematics and science (National Sciepce 
Foundation, 1997) . This is to be accompanied by 
development of community resources to sustain educatiopal 
improvements (Brown, 1996) . 



Program Improvement Review 



A primary means of accomplishing TURSI's aims is ^he 
Program Improvement Review. Done by 7URSI experts, 
typically retired teachers, the purpose of a Review is to 
identify strengths and weakenesses in schools' math and 
science programs and make recommendations for improvemen|t . 
ARSI experts, thereby, are charged with helping low- income, 
rural schools make students more productively employable in 
a science-based, technology- intensive world, in doing 
this, ARSI experts contribute to production of the hum^n 
capital needed for the economic and social development of 
rural areas. 



Purpose of the Pap er 



The following account is based on participant - 
observation done while I was first an ARSI expert -aspirant, 
then an ARSI expert doing his first Program Improvemei^t 
Review, and finally an ARSI dropout. The descriptions of 
"shadowing, " of neutral -site instruction, of report, 
preparation, and of rejection of the ARSI model are based 
on work done as part of the process of bringing ARSI to^ 
West Virginia under the auspices of the regional university 
where I am employed. Participation in this process le^ds 
to the following observations: 



ARSI experts construe the process of 
educational achievement as a thorougjhly 
understood, relatively simple mechanism 
manifest in static indicators of school 
effectiveness. 

In consequence, ARSI has standardized apd 
accelerated its Program Improvement Review 
process through excessively routinizep 
observation based on short-cut procedures 
and unvalidated instruments. 

ARSI experts uninterested in conventional means of 
substantiation for their evaluation criterfia, 
but take them for granted as embodying the 
one right way to teach math and science anywhere . 

Student responses and student -teacher 
interaction are irrelevant to ARSI evaluaticjns . 
Departures from ARSI criteria, even in the 
presence of overwhelmingly favorable student 
performance, are negatively sanctioned. 

The remainder of the paper is devoted to clarifying 
these observations based on my first-hand account of ARSI 
at work. Throughout, one important message seems clepr: 
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ARSI's Program Improvement Reviews in low- income, rura^ 
schools are likely to undercut achievement in science and 
math, rather than enhance it . 

We attribute this unfortunate set of circumstances to 
specious assumptions as to the existence of a science-based 
rationale for the routinization and streamlining of 
educational practice and evaluation. ARSI, in this v^ew, 
represents the persistence of Callahan's (1964) "cult of 
efficiency" at the turn of the millenium. 

A Checklist-Guided Audit 

The Program Improvement Review takes the form of a ope- 
day, one-expert school visit, yielding a checklist-guided 
audit, resulting in degree - of -complicance scores ranging 
from 1 to 5 on approximately seventy Likert items. The 
checklist is called a "Consistency Rating Summary. V 

For example, when evaluating a math program, the first 
general heading is "Curriculum, " subsuming ten Likprt 
items, the first being: 

x.l Tbe math cdcrjLcuJLum Xs writt^ ^nd 

used in planning the instructional 
program. 




5 



The remaining general headings are "Instruction," 
"Thinking Processes," "Equity and Diversity," "School 
Climate," "Relevance" or "Connections," "Training and 
Development," and "Financial and Material Resources. 

The total number of items varies slightly depending on 
the discipline, math or science, the grade level, and tl^e 
state which provides the educational policy setting for the 
review. Minor variations in the wording of the general 
headings and individual items are geared to these same 
factors. For example, under "Instruction," the first ilpem 
used in evaluating math programs in West Virginia 
elementary schools reads as followsv: 

2 . 1 Teachers use WV IGO ' s to guide their 

instructional practices. 

"WV IGO's" refers to state-mandated "Instructional 
Goals and Objectives," around which high-profile state 
achievement tests are organized. 

Likert item scores are used to gauge specif iq^ 
strengths and weaknesses in a school ' s math or science 
program. Strengths reflect consistency with the ARSI mojlel 
embedded in the "Consistency Rating Summaiy." Weaknesses 
reflect departures from the model. In practice, far m<pre 
attention is given to weaknesses than to strengths. 
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Source of the Checklist 



In spite of the importance of the Consistency Rating 
Summary, the source of its ten headings and seventy items 
is not identified. Are they research-based? Are t^ey 
reasonable inferences based on years of teaching 
experience? Are they established principles in math an(^ 
science education? Is their appeal based on face validity 
among ARSI experts? Do they represent an identifiable^ 
educational philosophy or pedagogical model? Participants 
are not tojLd. 



NSF Standards 



NSF has promulgated a detailed set pf National Sciejnce 
Education Standards (National Research Council, 1996) . In 
the course of conversation and training with ARSI expeij’ts, 
however, these are never mentioned. If the experts are 
aware of NSF Standards, they do not disclose this. If |TSF 
Standards are a source for the Consistency Rating Summary, 
participants are not told. 



Off-Handed References 



ARSI experts occasionally make off-handed references 
to "constructivism, " and they are fond of invoking the 
notion "hands-on." One might reasonably surmise, 
therefore, that these ideas, though they typically remain 
vague, are included in construction of the Consisteijicy 
Rating Summary and the way it is scored. 

State Mandates 

ARSI experts often refer to state mandates, such as 
West Virginia's Instructional Goals and Objectives, 
mentioned above, and the Kentucky Core Content fojr 
Assessment. Whatever the merit of these state-level 
mandates, their substance appears to have been anotjier 
influence in construction of the Consistency Rating 
Summary, and affects the way it is applied. The heading 
emblazoned at the topic of the Consistency Rating Summary 
may vary with the state in which it is being used, as in 
"KERActeristics of a Good Mathematics Program, " used in 
Kentucky, or the "West Virginia Program Improvement Rev|.ew 
Consistency Rating Summary for Mathematics . " 
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Brain- StormjLnq 



Beyond these tentative observations, however, no 
rationale for the instrument is provided. One is left with 
the impression that the Consistency Rating Summary may v^ry 
well have been the product of brain-storming sessions. The 
product is an instrument which sounds current and topically 
correct, but which, as an evaluation tool, is of uncertain 
value. 

Consistency Rating Summary Validation 



Similarly, the technical properties of the Consistency 
Rating Summary as a measurement tool are not reported, and 
may not have been investigated. Given organization of |:he 
instrument into ten sections, each subsuming six to ten 
items, one might reasonably surmise that a factor analypis 
would reveal ten identifiable sxibscales. If this is the 
case, however, results are not available. The same is tpue 
for routine reliability coefficients. In short, the 
psychometric properties of the instrument seem not to bp 
known. The possibility that discussion of such properties 
might be pertinent, even essential, is not acknowledged by 
ARSI experts. 



Re porting on a Program Improvement Review 



The final report, usually written overnight an<^ 
presented the next day, is organized around the same ten 
general headings and seventy Likert items . Since much m|Ore 
attention is given to weaknesses than to strengths, most 
reports do not address all general headings or all it^ms, 
but only those deemed deficient . 

Recommendations for change appear throughout |:he 
report. A recommendation pertaining to "Releveance" , 
meaning "[relating] mathematical knowledge to students V 
goals and interests," for a middle school located in a low- 
income, rural county in West Virginia's southern coal 
fields reads as follows: 

"Make a concerted effort to display positive, 
engaging images of mathematics throughout the 
school environment, paying particular attention 
to highlighting student work that is creative 
(not just correct) [Empahsis in the original.] 
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Becoming an ARSI Expert 



Training in doing the Program Improvement Review, 
including scoring the Consistency Rating Summary, usually 
begins with "shadowing," accompanying an ARSI math oi^ 
science expert who is doing a Review. ARSI experts also 
provide training at neutral sites, relying heavily on 
videos prepared to meet their specific instructional 
needs. Limited role-playing is used as a means of readyjLng 
prospective experts to present Program Improvement Review 
findings to school personnel. 

Training is informal, with little or no direct 
instiruction . Instead, the ARSI experts, most often retired 
teachers, serve as models during shadowing, and provide 
illustrative opportunities to apply the ARSI model during 
training sessions. 

Total training time varies, usually ranging from (pne 
to two days. An ARSI expert may also participate in the 
first Program Improvement Review done by a just-trained 
expert . 



"Shadowing” in Chemistry 8-B: Deficien t Instruction 



To illustrate our claim that ARSI Program Improvemept 
Reviews are likely to undercut achievement, we begin with a 
brief case study of shadowing. Two ARSI expert -aspirapts 
are observing the in- school work of on ARSI science expert 
at a small, rural, low- income elementary school in eastern 
Kentucky . 

We first attend a chemistry class. The three of up 
open the front door to the classroom without knocking, walk 
to the rear without speaking, and sit in side-by-side 
desks, while the class goes on about us. Students seem 
uninterested in our intrusion. The teacher seems 
unconcerned, and she makes no effort to acknoweldge our 
presence . 

Even though this elementary school goes through grpde 
8, chemistry, rather than, say, general science, seems out 
of place, too advanced for an elementary school. The^ 
class, moreover, is referred to as Chemistry 8-B. This, we 
learn, means that chemistry students are grouped ox^ 
tracked, with the ostensibly more capable students located 
in section 8|-A. 
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Nevertheless, the twenty-five students in section 
seem quite capable themselves. The teacher is reviewing 
chemical bonding, referring to positive and negati'v^e 
valences, what they mean with regard to the make-up of 
individual atoms, and how they govern the way differei^t 
elements combine to form molecules. She makes occasional 
reference to a periodic table displayed within easy re^ch 
on the wall near the front of the room. 



Desks are organized in traditional fashion, arranged 
in six rows, all facing forward. The teacher's desk is in 
the front of the room in the middle, turned toward th^ 
students. The teacher stands slightly to the left of her 
desk facing the students and occasionally turning to phe 
board or, less often, to the periodic table. 



The presentation, too, is traditional, relying largply 
on lecture and board work, with questions and responses to 
teachers' queries from students. The teacher speaks faiply 
rapidly. The substance of the class is in no sense 
trivialized to match the ostensibly limited capabilities of 
lower track students . 

The material covered is high school chemistry much as 
I remember it from the eleventh grade. The teacher, 
though, is smarter and more articulate, explaining things 
more clearly than mine did decades ago. Her high 
expectations for students are genuinely taken for granted. 
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None of the students stands out as a stellar perforyner 
or favorite. The teachers' high expectations seem to apply 
equally to everypne. 

The truly remarkable things about the class are tljie 
students' responses. All white, about half male and half 
female, they seem genuinely engaged. They attend single- 
mindedly to the teacher's presentation. 



The students, manifestly, are putting all their t^me 
on task. Not just any task, but the conceptually 
difficult, even esoteric task at hand^ 

The teacher asks questions fairly often. Answers are 
quickly forthcoming, spoken thoughtfully, usual 3^y 
confidently, without the formality of hand- raising. 
Students' questions are immediately acknowledged a^d 
answered in a business-like, though not unsympathetic 
fashion. 



The teacher, a woman of about thirty who seems^ 
obviously to enjoy what she is doing, tries various means 
of explaining the same difficult ideas, sometimes, 
complementing her oral presentation with additional board 
w^rk. 




Students don't talk among themselves. Two girls on 
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the teacher's right near the front of the room are an 
exception, but as they whisper, they look toward the chalk- 
board, and one points to a diagram that the teacher had 
drawn earlier, illustrating the bonding of sodium ^nd 
chlorine . 

A male student near the rear of the room on the. 
teacher's left has a persistent problem with understanding 
her explanation of positive and negative valences. He 
makes his difficulty conversationally evident: 

"Yeah, but I still don't get it. The signs are phe 

opposite . . . . " 

He makes his point, in the same conversatiojial 
fashion, more than once: 

"I still don't get it. Why isn't it negative ....?" 

The teacher explains again, varying her choice of 
words. She gives no evidence of impatience. She addrespes 
the questioning student in a matter-of-fact, even collegial 
fashion. 




She moves on, still holding students' attention, apd 
doing so effortlessly. She presents material with relaxed 
enthusiasm born of genuine interest. There is na. 
exaggerated affect or undue dramatization as she continues 

15 



18 



with a traditional presentation of conceptuaJLly 
sophisticated material . 

The puzzled student on the teachers' left remain!^ 
confused about positive and negative valences, though the 
precise nature of his misunderstanding is still not qu|te 
clear. He remains engaged, however, and raises the issue 
yet again, without evidence of embarrassment or anxietj^. 
The teacher stops and thinks, looks at her diagrams on the 
board, seems not to know what else to say^ 

A male student sitting to the immediate left of his 
confused colleague responds spontaneously and matter-of- 
factly : 



"I think I see ... try this.*? 

I cannot hear what is said. After a brief exchange 
between the two students, the puzzled one addresses ^he 
teacher: 



"If sodium is short an electron and it adds c^ne, 
why isn't it negative?" 



Implied in this question is a complementry query about 
chlorine: if chlorine has an extra electron and it gives 

one to sodium, why isn't chlorine positiye? 
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The source of the student's confusion is now clear. 

The +1 valence of sodium is determined in its free sta|:e, 
before it combines with chlorine to form table salt . The 
fact that it takes an electron from chlorine -- 
in effect adds a negatively charged particle -- does not 
make it negative. The fact that it has a place for an, 
electron that it adds to its outer ring, however, does make 
it positive. And conversely with chlorine. 

The nature of the difficulty having finally been 
clarified, the teacher is able to disspell the formerly 
puzzled students' misunderstanding. He is satisfied. 

The teacher and students continue in the same matter- 
of-fact but enganged manner which has prevailed from the 
beginning. One way to usefully characterize their approach 
and the nature of the affect which accompanies it might 
very well be "professional.'! 

As an observer, I was stunned. How did the teacher 
manage to hold the attention and active interest of this B- 
level -- or any level -- eighth grade -- or any grade 
chemistry class -- or any class -- throughout an entire^ 
period in which she discussed, in traditional lecture form, 
chemical bonding? In a low-income, rural, K-8 elementary 
school in eastern Kentucky or anywhere else! 
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Here, as best I could determine, was science being 
taught and learned about as well as could be done. Sinpe 
the aim of ARSI is to promote high achievement in math and 
science in low- income, rural schools, this, perhaps, wap a 
model, though one that might prove difficult to codify. 

NSF's National Science Education Standards, which piay 
or may not be known to ARSI experts, are intended to enable 
educators to judge whether particular actions will seifve 
the vision of a scientifically literate society (National 
Research Council, 1996) . The actions of this teacher ^nd 
her students emphatically did just this. Or so it seemed 
to me . 

An Off -the -Cuff Evaluation of Chemistry 8-E 

At the behest of the ARSI science expert, the three of 
us who had been observing left before the class was ovpr. 

We had been in Chemistry 8-B for about twenty- five 
minutes . 

Going out the door at the front of the room, I said to 
the teacher: 

"We're leavin' 'cause we can't understand tjiis 
stuff. " 
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The teacher stopped in mid- lecture, looked at me while 
I was speaking, and an expression of uncertainty left her 
face as she smiled. She gestured toward her students ^nd 
said with confident pride: 

" They can understand it.*'" 

"I can see that!", I replied, as I joined the other 
two observers in the h^ll . 

As the three of us walked to the next class, the A^RSI 
science expert, striding purposefully, leading the way, 
offered the following judgments .r 

"They didn't understand a word she said." His tone 
was contemptuous. "She was way, way over thpir 
heads . " 

"There was nothing to hold their interest, no 
manipulatives or anything." 

"The walls were just about bare. Not much abput 
science on them . . . nothing at all about science 
careers . 

"She was traditional lecture the whole time. ^11 



content . " 



The other observer was non-committal, as if taking in 
what was being said but still processing it, neit^ier 
concurring nor disagreeing. 

There was a brief silence as we walked. Then I s^id, 
laughingly, "for what it's worth, she teaches just like I 
do, when I'm having a really good day." Neither the A^RSI 
science expert nor the other observer acknowledged my 
comment. The ARSI expert led us into the next classroom. 

I felt sort of silly. Not because no one had 
acknowledged my response, but because I had felt the n^ed 
to cover it with self-deprecating laughter. 

It was clear that the ARSI expert had definite 
preconceptions as to what eighth graders could and could 
not handle. His conclusion that the students in 8-S^ 
chemistry had no idea what the teacher was talking about 
seemed wildly at odds with what I had seen and heard in the 
classroom. 



Even the puzzled student eventually understood, and he 
did so with the help of another student. His confusion, 
moreover, bespoke an understandable, even imaginatij/e, 
failure to see the specific terminological conventions 
which were being employed. In a real sense, his confusjLon 
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about teminological conventions actually reflected a clear 
understanding of the chemical bonding process itself. 



The teacher's method of presenting the material wai^ 
traditional, to be sure. The students participated freely, 
however, without fear and without required hand-raising. 
The teacher- student , and student -student exchanges were 
conversational and matter-of -factly animated. Studejits 
helped each other. 

The_JLcreleveance of Students 

For the ARSI expert, however, this sort of relaxed, 
informally traditional approach was inevitably 
ineffective. It seemed abundantly clear that the livin«j 
presence of students in the classroom was not essential to 
his judcrmenhs . He seemed not to notice them, their 
engagement, or the informed nature of their exchanges with 
the teacher and with each other. The expert attended Ojily 
to the teacher, her traditionally limited use of few 
instructional materials, and the dearth of wall 



One Best Wav to Teach Science 

The ARSI expert clearly judged himself to be i^ a 




position to evaluate any science teacher's performance 
without benefit of observing or otherwise evaluating 



student responses, to which he seemed oblivious. In this 
instance, he purported to know in a matter of minutes tjhat 
the teacher was clue -less, and that students would not 
learn. Traditional lecture was bad. Absence of\ 
manipulatives was worse. "You can use them to build 
molecules," he assured us. 

"That's what she was trying to do, but it's 
something you have to get your hands on. Thpre 
weren't even any [manipulatives] in the room." 

Thin Descriptioji 

/ 

The ARSI science experts' dismissive, almost an^ry 
assessment of the teacher's effectiveness bespoke a 
willingness to generalize from very limited informatipn. 

His assumption, clearly, was that twenty-five minutes of 
haphazardly selected, barged-in-on class time enabled jiim 
to produce an accurate typification of the teachers' 
pef ormance and students ' consequent achievement . 

His harsh judgments, moreover, seemed inconsistent 
with NSF's position that science teaching and inquiry cap 
be effectively done in a variety of ways (National Research 
Council, 1996; also see National Science Teachers^ 
Association, 1998) . But once again, the connection between 
NSF and ARSI may or may not entail a shared understanding 
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about teaching science and math. NSF standards may not be 
known to ARSI experts, 

ILShadowinq" in a Program Improvement Review Presentation 

Three weeks after the visit to the eastern Kentucky K- 
8 elementary school, I was again involved as an observe^. 

I was paired with the same ARSI expert -aspirant , shadowing 
another ARSI expert in another small school . Also Ipw- 
income and rural, this eastern Kentucky K-6 elementary 
school was located about fifty miles from the school in 
which we had observed Chemistry 8-B. 

During an hour-long, late -morning meeting, the A^SI 
expert presented his previous-day ' s findings to the 
school's principal and six teachers. The ARSI expert be^an 
with a weak, almost apologetic grin: 

"This isn't as bad as it looks. There are a lot of 

I's, 2's, and 3's, but this can be fixed ... a lot 

of it ...." [His voice trailed off. I 

Criteria used in selecting the six teachers present at 
the meeting were not specified. They and the principaij., 
however, remained silent as the ARSI expert went over his 
largely unfavorable report^. 
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"There's no evidence of the importance of math. 

They come away thinking it's just what they do in 
school . " 

"They don't create their own knowledge. There ip a 
lot of mainly lecture in the classrooms . " 

"If you used field trips, they would be able to 
see math all around us . " 

"They don't see its importance for careers, ^nd 
that it ' s rewarded . " 

The principal, in spite of the beating her school was 
taking, looked confident and even eager throughout, as if 
to say "we're professional educators sharing informatipn. 
There's nothing personal about this. We're glad to hear 
from outside experts, and we'll benefit from it. Please go 
on. " 



The teachers seemed affectively disengaged bul^ 
dutifully attentive. Their faces betrayed no emotion. 

They seemed to neither accept nor reject the ARSI expert's 
account . 
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Teachers ' Informal Challenge 



After the report was presented, with only a few words 
of perfunctory discussion, we went to lunch in the school ' s 
cafeteria. By chance, I stood in the chow line with two of 
the teachers who had been present at the meeting. 



Female, white, in their late-forties to mid-fifties, 
the teachers pleasantly initated a conversation by ask|.ng 
where I was from. We talked briefly about West Virginia 
and work I had done in a rural county there. I likene(^ 
that to what was being done by ARSI in their school. This 
was followed by a brief what -do-we- say-now sort of 
silence. 



By way of keeping the conversation going, I added tpat 
the West Virginia project had been a long one. One of the 
teachers asked how long. I replied that it had gone on for 
three years, relying heavily on repeated focus groups with 
a broad range of stakeholders, and on literally hundreds of 
visits to the three schools involved. 



The teachers became more animated and emphatl^c. 
Speaking of the ARSI expert's report of instructional 
omissions and other deficiencies, they commented: 
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"We do a lot of that stuff, but we don't do it 
all the time. He was only here for one day, for a 
few hours . . . . " 

"I know teaching's not a 9 to 5 job, but there's 
only so much you can do . " 

"He never came to my room. How could he know wpat 
we do?" 

" I never even knew he was here . 

Clearly, in this low- income, rural elementary school 
in eastern Kentucky, the implied claim of 

representativeness made by the ARSI expert in our previous 
Kentucky visit was being challenged by teachers as simply 
wrong. This claim, nevertheless, tacitly undergirds all 
AJRSI Program Improvement Reviews. 

J 

In retrospect, it seems obvious that I invited this 
challenge from the two teachers with my mention of a thrpe- 
year project in West Virginia. At the time, however, I was 
just awkwardly trying to hold up my end of a conversatiop. 
Moreover, the teachers' responses seemed genuine, something 
they were waiting for a chance to say. Perhaps I had giyen 
them a deserved rhetorical opportunity, rather than a naked 
invitation to engage in a defensive, self-serving polemic. 
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Fiyjng pe.gi,ciengj.gs : Conservation of Momentum 

In addition to shadowing, the training of ARSI expert- 
aspirants includes neutral-site instruction offered by A^SI 
experts. As an example, ten ARSI expert -aspirants met with 
an ARSI math and science expert at the small -city 
headquarters of a West Virginia regional education agency. 
It was the ARSI expert's aim to continue with thev 
introduction of expert -aspirants to the ARSI approach in 
evaluating program quality. 

ARSI Training Videos 

A retired teacher, the expert relied largely oip a 
series of videos intended to provide opportunities to 
illustrate the ARSI ethos in use. 



During one of the longer and more purposeful videos, a 
white female teacher in her late twenties is seen reviewing 
the concept "conservation of momentum" with her eleventh 
grade physics class . There are approximately twenty 
students, all of them are white, about evenly divided 
between males and females . 
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Is this a functioning classroom, or something staged 
by ARSI to aid in the production of new ARSI science pnd 
math experts? We are not told. 

The students in the video are more or less attenti^ve. 
The teacher's presentation is brief and seems to lack 
focus, perhaps because the video begins near the end of her 
explanation, immediately following an exercise with 
manipulatives . 

Oddly, there is no teacher's introduction to the vicjieo 
itself. It just starts. Whether or not this is an ARSI- 
staged video, the absence of an introduction, an. 
explanation to students as to why the video is being shown, 
is disconcerting. After all, we are supposed to be engaged 
in the evaluation of instruction. Maybe the ARSI expert 
will use the teacher's failure to introduce her video a^ a 
painfully obvious illustration of the wrong way to do 
things, such as use audio-vidual aids in explaining 
conservation of momentum. 

The video is devoted entirely to cars crashing. Cprs 
crashing into each other, cars crashing into telephone 
poles, cars careening off guard rails and rolling ontov 
their roofs, cars going off the road and landing in ditches 
.... It is reminiscent of a demolition derby, but withput 
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a winner. It is not immediately evident to me that t|he 
video actually does illustrate conservation of momentum. 

The ARSI expert says nothing. The only sounds in (pur 
room, as in the classroom on the video, are made by 
crashing cars. 



As we watch the students watching the video, thej^ 
seem, for the most part, unmoved. The camera catches two 
male students sitting together laughing at one, seemingly 
unexceptional collision. 



The crashes, presumably, were staged. All the cars pire 
from the middle and late '70's. The video is repetitious, 
it seems too long, there is no narrative, just wreck af|:er 
wreck, one looking more or less like another. 



Finally it occurs to me that conservation of momentum, 
as best I can remember from twelfth grade physics, is 
manifest in the cars' tendency to continue moving eyen 
after they run into something solid. Though this 
recollection, in retrospect, seems embarrassingly obvi<pus, 
is it safe to assume that the students on the video made 
the same inference? After all, their teacher, much as o|ir 
ARSI expert, provided no commentary. Is this an example of 
constructivism, of students constructing their own physical 
knowledge? 
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When the conservation of momentum video is over, our 
video is over, too. If there was an in-class discussion of 
what the students had just seen, we didn't get to hear it. 
Employment of the video seems part of a badly disjoin|:ed 
instructional process. 

Perhaps the point of all this has been self-evidpnt 
to the other ARSI expert -aspirants . I am, however, 
surrounded by nine other adults, all involved in, education 
in one way or another, some with backgrounds in science, 
but more from administration or higher education. I wonder 
how many know what conservation of momentum means. Even 
now, I'm not sure that I do. For all I know, my afore- 
mentioned recollection from twelfth grade physics was in 
error. After all, I may have confused conservation pf 
momentum with "objects in motion tend to stay in motion 
....", or something like that. 

I wonder how many of the others see the pertinence of 
a video of serial collisions to understanding conservatjlon 
of momentum. Were they able to recall or construct their 
own physical knowledge? Or is this video as bad an 
instructional tool as it seems to be? 
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The ARSI expert has very little to say about, the^ 
serial-collision video. For a moment, he seems at a loss. 
He passes up the opportunity to fault the teacher for ^ot 
providing an introduction. He says nothing about the 
absence of a debrief 4.ng. 

Then, belatedly, he calls our attention to the fpct 
that two male students had laughed: 

"You could see their interest. They weren't jjist 

being passive . " 

The expert says nothing more about the video. He |ias 
concluded, as far as I can tell, that it demonstrated 
students' engagement in the process of acquiring a clear[er, 
deeper understanding of "conservation of momentum." 

Perhaps we really have seen the construction of physical 
knowledge . 

My colleagues and I are silent. In truth, the serial 
collision video seemed like a silly caricature of 
instruction with audio-visual aids, how to misuse t|iem 
rather than use them. But the ARSI expert gives no 
evidence of sharing this viejw. 
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Training in Fixin g Deficiencies : Getting "Down and Dirty” 



In another instructional video, a white female teacjier 
in her early thirties is standing in front of a class of 
elementary school students. We are not told the grade, |Dut 
the children appear to be eight or nine years old. Once 
again, all the students are white. 

The classroom is organized in traditional fashion, 
with individual desks in rows and the teacher standing at 
the front of the room, her back to the chalkboard. The 
teacher has said only a few words, the point of her clasps 
has not yet become evident, when the ARSI expert interrupts 
while the video continues to run. He speaks emphatically 
and with excitement : 



"Look at her! Look at her clothes! She prepared 
for this ! " 

In truth, I saw nothing distinctive about ^he 
teacher's clothing or appearance. She was dressed 
modestly, wearing a buttonless jacket with lapels, a whjLte 
blouse which buttoned at the neck, a just-below-the-knees 
skirt, and shoes with medium heels. Her clothing was wejLl- 
suited to working as, say, a bank teller, a receptionist in 
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a family dentist's office, or a casework supervisor ip. a 
state social welfare agency. 

Her hair was cut short, but not excessively so. :i^t 
was neatly combed, but not stylishly done. She wore make- 
up, but there was nothing ostentatious or extraordinary^ 
about it. She looked like the girl next door, grown up and 
working for a modest living. 

But the ARSI expert did not see it that way. The fact 
that the teacher was presentable counted against her: 

"She can't get down-and-dirty dressed like that." 

"She didn't come to work.v 

These observations, coupled with his surmise that the 
teacher had come prepared to appear on a video, seemed to 
imbue the ARSI expert with a sense of discovery. His 
, response to the video suggested that, perhaps, he had n^ot 
seen it before. He was looking for something instructive, 
and quickly found it in the teacher's appearance, which 
still seemed unexceptional. 

He judged a teacher's work as inevitably involving 
getting "down-and-dirty." Suitable clothes, I concluded, 
would have been faded jeans, a sweatshirt with holes wprn 
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in the elbows, and grass-stained tennis shoes. Why 
suitable attire for an elementary school teacher should^ 
take this form remained a mystery to me, just as the nature 
of "getting down and dirty" and why it was a pedagogical 
essential remained unexplained. 



None of the prospective experts spoke. I saw two gjive 
obligatory grins at the "she can't get down-and-dirty 
dressed like that" judgment. Otherwise, the group was, 
impenetrably difficult to read. Was the lesson clear? Did 
participants accept it? Did anyone find this, 
informative? Did the ARSI expert know that NSF National 
Science Education Standards do not include a dress code^ 
(Still again, was he aware that NSF has Standards?) Was he 
aware that teachers' attire is often an issue in rupral 
Appalachian schools because they sometimes dress too 
informally? Is this what it means to become an A|ISI 
expert? 



ARSI in West Virginia 

ARSI's first Program Improvement Review in West, 
Virginia was done in mid-March of 1999 . This was also the 
first time I worked as an ARSI expert. The same was trpe 
of my shadowing partner, who was serving as coordinator of 
our three-school review. Though newly-minted as an AjlSI 
expert, he had long experience in grant writing, program 



development, and administration of ground-up educatiopal 
change efforts. Early in his career, he had taught high 
school science. 

AdaotatioTLor Adoptian 

This Review, moreover, was to be different. It^ 
involved three schools rather than one. The schools, an 
elementary school, a middle school, and a high school, ^re 
in close geographical proximity to each other, situated in 
a low- income, rural county in the state's southern coal 
fields. 



In addition, while the ARSI Program Improvement Review 
was being used as a point of departure, it was not a 
governing model. The Consistency Rating Summary, replete 
with Likert items, was still there, but as only one source 
of information in preparation of a report which was to be 
tentative, formative, and qualitative. 



Rather than one-expert school visits, as in Kentucjky, 
there were four evaluators for each school . Most members 
of each team were newly-minted ARSI experts, who also pad 
training and experience in a variety of areas, including 
assessment, math education, program evaluation, pnd 
administration . 
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Recommendations for improvement were to be made only 
after discussing the final report with a variety of lopal 
stakeholders from the three schools. Stakeholders would 
participate in the process of actually producing ^he 
recommendations . 

Synthesizing^^ Final Report. 

My primary task was to synthesize a final report. 

The Consistency Rating Summary would have left little t<^ 
synthesize, but its place was not central in West Virginia, 
as it had been in Kentucky. The materials provided por 
synthesis were submitted in manila folders, eleven-by- 
seventeen envelopes, three-ring notebooks, translucent zip- 
lock packets, paper-clipped pages, and a spiral tablet. 

Consistency Rating Summaries prepared by ARSI experts 
were included. The Summaries, however, were mixed in with 
field notes, handwritten reminders, and miscellaneoyis 
jottings on single sheets of paper. In addition, each 
teacher at each school had completed a Consistency RatjLng 
Summary, and these, too, had been included. 
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Some ARSI experts' Summaries had conspicuous margipal 
notes and some did not . Summaries for the same school 
included and excluded different headings and items. S<pme 
included experts ' names and some did not . Some had a 
formal, finished appearance, while others looked l^ke 
preliminary worksheets. 

In spite of our plan to make production of, 
recommendations a collaborative effort with stakeholders, a 
few Su^aries included recommendations. All tolled,^ 
however, the materials did not resemble the output of the 
sort of mechanically routinized process of tjiin 
descriptions we had seen in Kentucky. 

A Forma tij?e Svsbemic Report 

Since our Review involved three schools, a systemic 
report seemed in order. Furthermore, even though tl^e 
schools were at three different levels, dramatic cross- 
school commonalities in traditional educational philosophy 
and old fashioned, no-nonsense practice made a single 
report seem fitting. 

The flexibly formative nature of the process was, 
acknowledged in the report ' s opening paragraphs under the 
heading "Informed Interpretation from Multiple 
Perspectives" : 
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"A good deal of what we have to say, moreover, 
is subject to good- faith interpretation ^nd 
re -interpretation by stakeholcjers 

Similarly, use of the Consistency Rating Summary yas 
placed in context, subsumed by "Judicuous Use of a 
Quantitative Rating Summary" : 



" . . . but one source of information for making 
formative judgments. Its ... scores ... mere!|.y 
summarize some of the information used in making 
our essentially qualitative judgments." 



First Dy^t 



The report characterized the math program in each of 
the schools as traditional, and noted that all adult 
stakeholders, teachers, administrators, and parents, 
preferred it that way. Parents were unaware of 
alternatives. Even some of the teachers were unfamiliai^ 
with current terminology and practice. When a newly-minted 
ARSI expert who teaches math education used the ternj^ 
"rubric" in a discussion with one elementary teacher, the 
teacher asked what rubric mea|nt. 



The schools were autonomous to a fault. Though 
constituting a rudimentary feeder system, teachers and 
administrators had no cross-school contact. Insofar ap 
their math curricula were cumulatively compatible, it was 
due to state and district requirements, and adherence to 
the same traditional ethos and practices. 

The report went on for twenty- seven double- spaced^ 
pages, addressing topics such as "Avoidance of Innovation, 
"Cautious Selectivity, " "Exclusion of Exploration, V' 
Innovations Come and Go," "Traditional Parental Roles," 
"School -to- School Isolation, " and "Staff Development ^nd 
Teachers' Traditionalism." 



The concluding sections re-emphasized the importajice 
of understanding the report as interpretative, formative, 
and subject to legitimate challenge by stakeholders. 
Readers were reminded that formulation of recommendations 
was to be a collaborative effort. 



"Their Nickel" 

When I gave the report to my former shadowing partn|er 
still coordinating this first West Virginia Review, his 
response took me by surprise. Noting the absence ojC a 
"Consistency Rating Summary," he said, "it's their 
nickel." In short, whatever liberties we took with the 
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ARSI model, this remained an ARSI endeavor. ARSI was 
establishing itself in West Virginia under the 
institutional auspices of our regional university, and some 
ARSI expectations had to be met . 

I used the diverse, unstandardized information which 
had been submitted, and tried to synthesize a set of 
defensible Likert item scores for the three- school system. 
Having attached this to the narrative, I thought the job 
was done. 



The coordinator agreed. He submitted a copy to Wesj: 
Virginia's first ARSI Collaborative Director, and scheduled 
a meeting^ 



Meeting with ARSI Officials 

The meeting with the ARSI Collaborative Director an^ 
an associate began amicably. They had read the report, and 
they listened with what appeared to be friendly interest as 
we briefly explained our plans to meet with stakeholders 
from the three schools to collaboratively prodyce 
recommendations for change. 



I characterized the approach to Program Improvement 
Reviews I had seen in Kentucky as "take-it-or-leave-it , " 

" expert- centered, " "prematurely codified," and "quick-and- 
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dirty." The evolving West Virginia approach to achiev' 
ARSI's objectives was, by sharp contrast, "flexible," 
"client-centered," "qualitatively formative," ^nd 
"collaborative . " 

The Director responded by noting that there was oi|ily 
one Consistency Rating Summary for three schools . 

I rep li eel: 

"Right. Like we said in the report, we too|c a 
systemic approach. It made sense, especially 
since the schools are so much alike. 

The Director responded that there were no 
recommendations. I referred again to the report, not;Lng 
that the recommendations were to be produced 
collaboratively with school-level stakeholders. 

The Director, still smiling, shook her head. She 

said: 

"The reports are standard. We need Summary scores 
for each school, and recommendations for each." 
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I responded that I had seen take-it-or-leave-it 
reports, loaded with misguded Likert -item claims tq 
precision, done all too quickly during shadowing. They 
were the sorts of reports, I added, that later sat on 
shelves gathreing dust, because stakeholders were not 
involved in their production. 

The Director replied: 

"I'm sorry if that was your experience." 

She looked at her assistant and asked: 

"Is that the way you saw it when you made visitp?" 

The assistant shook her head and murmured 
unintelligibly. 

I returned to my characterization of what I had s^en 
in Kentucky, including again "take-it-or-leave-it," 
"prematurely codified," and "quick-and-dirt^. " 

The Director responded: 

"But that's just your opinicjn." 

I snapped angrily: 




42 



45 



"Of course! What else would it be?" 



My shadowing partner intervened. He asserted that he 
had not expected to do Program Improvement Reviews exactly 
as they were done in Kentucky. He was especially concerped 
about formulating recommendations without collaboration 
with local stakeholders . 

"They need to be involved in this process. Thpy 
need a sense of ownership. Otherwise, the report 
will never be implemented.'? 

The ARSI Collborative Director was not persuaded. She 
said little, remained unflappable, and would not bud,ge: 

ARSI Program Improvement Reviews were standard. 

I asked: 

"What did you think of the text of the report?" 

The Director and her assistant both nodded approval . 
Then the assistant adcjed: 

"It was long. People are busy ...." (Followed b|^ a 
good-humored, partly muffled chuckle.) 
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I asked: 



"What's missing from the report as it is no|«??" 

The Director repeated that Consistency Rating 
Summaries and recommendations for each school were 
essential parts of any ARSI Program Improvement Review 
report. These, in fact, as submitted by the ARSI experts, 
are the report . 

I respondeji: 

"So I just clip the three reports together? It's 
a clerical j ob? ! " 

The Director replaced: 

"Yes ... in part . " 

I responded angrily: 

"If I had known we were gonna do it this way, 

I'd never have gotten involved. This is the Ipst 
one I'll do." 
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By this time, I had lost my composure, while tha 
Director had retained hers. I left the room acknowledging 
that ARSI would get the kind of report it wanted^ 

That was the end of my involvement with Appalachian 
Rural Systemic Initiati^ve. 

In Retrospect 

It is worth noting that, until our Program Improvement 
Review, ARSI had kept a low profile in West Virginia. 
Unknown to us was a series of three meetings with West^ 
Virginia educators hosted by ARSI representatives, one of 
whom is now the ARSI Collaborative Director. 

According to a participant, a former math teacher who 
is currently a professor of education, the meetings wpre 
held January through April of 1998 . Her unsolicited 
invitation to attend described the first meeting as^ 
intended to explore "the development of a self-assessment 
instrument ... to aid counties in: 

"Critically looking at their science and math programs." 
"Analyzing test data to improve instruction/curriculijm. " 
"Planning Professional Development." 
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"Revising unified Plains." 



However, the meeting actually focused on Kentuc|cy- 
style Program Improvement Reviews organized around 
Consistency Rating Summaries. 

The ARSI representatives' first question to all 
present suggested real openness: "Should we do this at ^11 

in West Virginia." As the participant's account goes, 
however, "my impression was that the decision had already 
been made." It would be done, and in the same way as in 
Kentucky . 



A math and science expert from ARSI's Kentucky program 
presented his state's version of the Consistency Rating 
Summary for review. Abandoning the "should we do this at 
all in West Virginia?" attention-getter, ARSI 
representatives instructed participants to examine epch 
item on the Summary according to unspecified national 
standards and state -mandated West Virginia regulations. 
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In a February 10th e-mail from the West Virginia, 
Collaborative Director, participants were told they "should 
decide if a question should be kept as is, deleted, pr 
modified. You may also generate additional questions." 
Latitude, yes, but narrowly circumscribed by production of 
Likert items. 

Reservations as to whether and how Program Improvement 
Reviews should be done in West Virginia had been 
disspelled. The Kentucky model, giving pride of place to 
the Consistency Rating Summary and check- list guided 
audits, had been adoptpd. 

Conclusion 

When we first thought about this paper, our main^ 
concern was that there is nothing rural about the work of 
the Appalachian Rural Systemic Initiative. That remajins 
emphatically true. For ARSI, a-school-is-a-school-is-a- 
school . 

We concluded, however, that even more important thc^n 
ARSI's failure to find anything consequentially distinctive 
about rural education was the characterization we offered 
at the beginning; 
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ARSI experts construe the process of 



educational achievement as a thorougjily 
understood, relatively simple mechanism 
manifest in static indicators of schpol 
effectiveness . 



In consequence, ARSI has standardized apd 
accelerated its Program Improvement Review 
process through excessively routinizep 
observation based on short-cut procedures 
and unvalidated instruments. 

ARSI experts,- however, remain unaware of 
substantiation for their evaluation criteria, 
but take them for granted as embodying the 
one right way to teach math and science anywheiqe . 

Student responses and student -teacher 
interaction are irrelevant to ARSI evaluations. 
Departures from ARSI criteria, even in the 
presence of overwhelmingly favorable studpnt 
performance, are negatively sanctioned. 



How has this come about? In his history of education 
classic. Education and the Cult of Efficiency . Raymond 
Callahan (1964) describes a set of public poljLcy 
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circumstances characterized by two assumptions, ope 
following from the other, First, and fundamentally, 
educational achievement is so thoroughly understood tpat 
its practice and evaluation can and should be routinized 
and streamlined. Second, and as an obvious corrollpry, 
well-informed rout inizat ion assures not only enhanced 
achievement, but provides opportunities for cost cutti^ng, 
as well . 



If these assumptions, especially the first, were true, 
educational reform, in low- income rural schools or 
elsewhere, could be inexpensively programmed and, 
monitored. Schools would become models of early- twentieth 
century, scientific management.. 

If the assumptions are false, however, science-based 
routinization gives way to to quick -and -dirty dogmatism, 
producing contemporary caricatures of Taylorism. 



As a result, as we have sought to make clear, ARSIVs 
Program Improvement Reviews in low- income, rural schools 
or any other schools -- are likely to undercut achievempnt 
in science and math, rather than enhance it. 




There may be merit to the claim that high levels of 
achievement in science and math are far more in^ortant 
today, for individuals and entire societies, than evep 
before. Whether or not this is the case, intervention by 
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the National Science Foundation's Appalachian Rural 
Systemic Initiative represents bad practice, indeed. 
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